[lxc-users] AWS EC2: timeout connecting to instance metadata webserver (169.254.169.254) for *some* URLs (when connecting from a LXD container)

[Tomasz Chmielewski]

That's a weird one!

In AWS, there is a concept of "instance metadata" - a webserver which 
lets you fetch some instance metadata using http:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html


For example, you can run this (from both AWS/EC2 instance and LXD 
container running inside a AWS/EC2 instance), it will return some 
metadata:

curl -v http://169.254.169.254/latest/meta-data/


Now, some of these requests time out when executed from a LXD container 
running inside a AWS/EC2 - but work perfectly from the very same AWS/EC2 
instance.


For example, this request works fine from AWS/EC2 instance (ignore the 
output - HTTP connection works just fine):

root at aws-instance:~# curl -v http://169.254.169.254/latest/api/token
*   Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 405 Not Allowed
< Allow: OPTIONS, PUT
< Content-Length: 0
< Date: Wed, 18 Nov 2020 22:41:46 GMT
< Server: EC2ws
< Connection: close
< Content-Type: text/plain
<
* Closing connection 0


However, when executed from within a LXD container running inside the 
very same AWS/EC2 instance - it times out!

root at lxd-container:~# curl -v http://169.254.169.254/latest/api/token
*   Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.58.0
> Accept: */*
> 


Even more weirdly, these work inside the container:

curl -v http://169.254.169.254/latest/api/
curl -v http://169.254.169.254/latest/api/t
curl -v http://169.254.169.254/latest/api/to
curl -v http://169.254.169.254/latest/api/tok
curl -v http://169.254.169.254/latest/api/toke


And this times out:

curl -v http://169.254.169.254/latest/api/token



Does anyone know why? tcpdump doesn't give me many clues (TTL?).


Tomasz Chmielewski