[lxc-users] AWS EC2: timeout connecting to instance metadata webserver (169.254.169.254) for *some* URLs (when connecting from a LXD container)

[Tomasz Chmielewski]

On 2020-11-18 23:50, Tomasz Chmielewski wrote:
> That's a weird one!
> 
> In AWS, there is a concept of "instance metadata" - a webserver which
> lets you fetch some instance metadata using http:
> 
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
> 
> 
> For example, you can run this (from both AWS/EC2 instance and LXD
> container running inside a AWS/EC2 instance), it will return some
> metadata:
> 
> curl -v http://169.254.169.254/latest/meta-data/
> 
> 
> Now, some of these requests time out when executed from a LXD
> container running inside a AWS/EC2 - but work perfectly from the very
> same AWS/EC2 instance.
> 
> 
> For example, this request works fine from AWS/EC2 instance (ignore the
> output - HTTP connection works just fine):
> 
> root at aws-instance:~# curl -v http://169.254.169.254/latest/api/token
> *   Trying 169.254.169.254...
> * TCP_NODELAY set
> * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
>> GET /latest/api/token HTTP/1.1
>> Host: 169.254.169.254
>> User-Agent: curl/7.58.0
>> Accept: */*
>> 
> < HTTP/1.1 405 Not Allowed
> < Allow: OPTIONS, PUT
> < Content-Length: 0
> < Date: Wed, 18 Nov 2020 22:41:46 GMT
> < Server: EC2ws
> < Connection: close
> < Content-Type: text/plain
> <
> * Closing connection 0
> 
> 
> However, when executed from within a LXD container running inside the
> very same AWS/EC2 instance - it times out!
> 
> root at lxd-container:~# curl -v http://169.254.169.254/latest/api/token
> *   Trying 169.254.169.254...
> * TCP_NODELAY set
> * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
>> GET /latest/api/token HTTP/1.1
>> Host: 169.254.169.254
>> User-Agent: curl/7.58.0
>> Accept: */*
>> 
> 
> 
> Even more weirdly, these work inside the container:
> 
> curl -v http://169.254.169.254/latest/api/
> curl -v http://169.254.169.254/latest/api/t
> curl -v http://169.254.169.254/latest/api/to
> curl -v http://169.254.169.254/latest/api/tok
> curl -v http://169.254.169.254/latest/api/toke
> 
> 
> And this times out:
> 
> curl -v http://169.254.169.254/latest/api/token
> 
> 
> 
> Does anyone know why? tcpdump doesn't give me many clues (TTL?).

A somewhat related post (with docker having a similar issue):

https://rtfm.co.ua/en/aws-eksctl-put-http-169-254-169-254-latest-api-token-net-http-request-canceled-2/


But, I'm no closer in getting a similar workaround for LXD.


Tomasz Chmielewski