Backend Engineer _ AI Gateway – Englewood Cliffs, NJ /Plano, TX

Job Title: Backend Engineer _ AI Gateway

 Englewood Cliffs, NJ /Plano, TX

Position Information                   

•            Contract Period: 1 yr.

•            Work Hours: 9-6 local time

•            Work Location:

-700 Sylvan Ave Englewood Cliffs, NJ 07632 (through September 2026)

-6625 Excellence Way, Plano, TX 75023 (effective October 2026 onward)

JD Details           We are building an AI Gateway — a high-performance, secure intermediary layer that routes, inspects, and governs all traffic between enterprise clients and LLM/AI service providers. This role requires deep expertise in network programming, async Python, and cloud infrastructure to design and operate a latency-sensitive, policy-driven gateway that handles massive concurrent streaming connections at scale.

What we are looking for

•            Async Python & Network Programming (3 + yrs) – Strong hands on experience with asyncio, aiohttp/anyio, low level TCP/UDP sockets, WebSocket & SSE servers, TLS handshake, connection pooling, keep alive, and zero copy buffering.

• Streaming & Large Payload Handling – Ability to manage chunked transfer, bidirectional streaming, back pressure and high throughput data paths for real time LLM inference.

•            Protocol Knowledge – Working familiarity with HTTP/2 & HTTP/3 (multiplexing, HPACK), gRPC, and occasional custom protocol parsers.

•            FastAPI Based Backend (3 + yrs) – Design, develop and test production grade REST/Streaming APIs; pragmatic use of Pydantic, dependency injection and OpenAPI docs.

•            Containerization & Kubernetes – Docker, AKS/EKS/GKE deployment experience; HPA/VPA scaling, rolling updates, and service mesh (e.g., Istio/Envoy) basics.

•            Async Messaging & Event Driven Design – Production use of Redis Streams, RabbitMQ or Kafka for decoupled task queues, retries and back off logic.

•            PostgreSQL & Vault – Advanced query tuning, partitioning, PgBouncer pooling, plus HashiCorp Vault (dynamic secrets, transit encryption) for key management.

•            Cloud Networking & Security – L4/L7 load balancing, TLS termination, reverse proxy (NGINX/Envoy), mTLS between services, and automated cert lifecycle (ACME/Let’s Encrypt).

•            IaC & CI/CD – Terraform modules for multi env infra, plus GitHub Actions/GitLab CI pipelines with container image scanning and canary/blue green deployments.

Work experience desired

•            Enterprise Security Gateways – Integration with CASB/DLP/WSS or similar edge security platforms; centralized logging & audit trails.

•            IAM & SSO – Configuring Azure AD/Entra ID, Okta, SAML 2.0 or OIDC SSO flows; managing Conditional Access & SCIM provisioning.

•            AI/LLM Pipelines – Building prompt engineering workflows, PII anonymisation, and monitoring Azure OpenAI (or comparable) usage, rate limits & cost.

•            API Gateway Operations – Custom webhook/REST integrations, circuit breaker patterns, rate limiting, auto retry and health checking.

•            Observability & FinOps – End to end monitoring with Datadog, Prometheus/Grafana or Azure Monitor; proactive cost optimisation and 24/7 incident response (SLA/SLO).