ai-engineering-interview-questions

AI Infrastructure and Scalability

• How do you improve inference speed in production LLM deployments?
  • Answer: LLM Inference Optimization
• LLM optimization techniques
  • Answer: LLM optimization techniques
• How do you select GPUs for LLM inference?
• What is model parallelism vs data parallelism in distributed training?
• What is tensor parallelism, and how does it help serve large models?
• What is pipeline parallelism?
• How does continuous batching improve LLM inference throughput?
  • Answer: Continuous Batching in LLMs
• What is speculative decoding, and how does it speed up inference?
  • Answer: Speculative Decoding
• What is KV cache, and how do you manage memory for it?
  • Answer: What is KV Cache in LLMs?
• What is Paged Attention?
  • Answer: Paged Attention in LLMs
• How does GGUF work?
  • Answer: How does GGUF work?
• How do you optimize inference for edge and mobile deployment?
• What is model quantization (INT8, INT4, FP16, BF16), and how does it affect quality?
  • Answer: Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization
• How do you implement auto-scaling for AI workloads?
• What is the role of load balancing in AI serving infrastructure?
• How do you manage GPU memory for serving multiple models?
• What is model sharding, and when would you use it?
• How do you implement request queuing and priority scheduling for AI services?
• What are the cost trade-offs between self-hosted and API-based AI inference?
• How do you handle cold start latency for serverless AI deployments?
• How do you implement model caching to reduce redundant computations?
• What is the difference between synchronous and asynchronous inference, and when do you use each?
• What is FSDP (Fully Sharded Data Parallel), and how does it differ from DeepSpeed ZeRO?
• How do you monitor and profile LLM inference in production (TTFT, inter-token latency, GPU utilization)?
• What is model routing at the infrastructure level, and how do you route requests based on complexity and cost?
  • Answer: LLM Routing

License

   Copyright (C) 2026 Outcome School

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.