Bug description
We are seeing memory usage typical for memory leaks on one of our apps with PHP 8.1 running and DDTrace versions 0.67.0, 0.68.2, 0.69.0 (we were not able to test 0.70+ due to reported issue https://github.com/DataDog/dd-trace-php/issues/1544) after upgrading from DDTrace 0.64.1 on long-running processes (mainly queue consumers).
In order to investigate further what is happening, we updated a second application (running PHP 8.0) also from DDTrace 0.64.1 to 0.69.0 and we can see non-negligible memory usage increase although I cannot say for sure it's a memory leak or just an increase in memory footprint since we're scaling the application a lot and therefore lifetime of a container is in matter of minutes, not hours.
I've seen that DDTrace started to use WeakRef/WeakMap on PHP 8.0+. I'm not sure if it's somehow related but we encountered issues with WeakRef that is reported and verified here: https://bugs.php.net/bug.php?id=81027.
We're also heavily using Guzzle in many of the consumers that now fail on memory.
I'd love to provide more information but I'm not sure how to debug ddtrace memory usage properly, especially when it exhibits only in long-running processes (I may try Blackfire but it's not easy to replicate the "traffic" locally).
I'm also not entirely sure it's DDTrace fault but as I said nothing had been changed on the second application except for DDTrace version and increased memory usage is clearly visible.
I'd appreciate any insight into what MIGHT have changed, any debugging tips and if an increased memory usage is expected or not.
Thank you!
(First application, showing average memory usage)
(Second application, nothing had been changed here except for DDTrace version, showing average memory usage, aggregated from multiple containers)
increased memory usage on our application running
PHP version
PHP 8.1.4 (cli) (built: Mar 20 2022 16:51:10) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.4, Copyright (c) Zend Technologies
with Zend OPcache v8.1.4, Copyright (c), by Zend Technologies
with blackfire v1.75.0~linux-x64-non_zts81, https://blackfire.io, by Blackfire
with ddtrace v0.71.0, Copyright Datadog, by Datadog
Tracer version
0.69.0
Installed extensions
[PHP Modules]
bcmath
blackfire
calendar
Core
ctype
curl
date
ddtrace
dom
exif
FFI
fileinfo
filter
ftp
gettext
hash
iconv
igbinary
imagick
imap
intl
json
libxml
mbstring
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
Phar
posix
readline
Reflection
session
shmop
SimpleXML
soap
sockets
sodium
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zip
zlib
[Zend Modules]
Zend OPcache
blackfire
ddtrace
OS info
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
Diagnostics and configuration
Output of phpinfo() (ddtrace >= 0.47.0)
{"date":"2022-03-21T14:27:48Z","os_name":"Linux f2f9e7dc8456 5.10.104-linuxkit #1 SMP Wed Mar 9 19:05:23 UTC 2022 x86_64","os_version":"5.10.104-linuxkit","version":"0.71.0","lang":"php","lang_version":"8.1.4","env":"olsavmic","enabled":true,"service":"testing-71","enabled_cli":true,"agent_url":"http:\/\/datadog:8126","debug":true,"analytics_enabled":false,"sample_rate":1,"sampling_rules":[],"tags":[],"service_mapping":[],"distributed_tracing_enabled":true,"priority_sampling_enabled":true,"dd_version":null,"architecture":"x86_64","sapi":"fpm-fcgi","datadog.trace.request_init_hook":"\/opt\/datadog-php\/dd-trace-sources\/bridge\/dd_wrap_autoloader.php","open_basedir_configured":false,"uri_fragment_regex":null,"uri_mapping_incoming":null,"uri_mapping_outgoing":null,"auto_flush_enabled":false,"generate_root_span":true,"http_client_split_by_domain":false,"measure_compile_time":true,"report_hostname_on_root_span":false,"traced_internal_functions":null,"auto_prepend_file_configured":false,"integrations_disabled":"default","enabled_from_env":true,"opcache.file_cache":null}
Diagnostic checks | passed
Upgrading info
Upgraded from DDTrace 0.64.1 -> 0.67.0, 0.68.2, 0.69.0, all versions are showing the same behaviour.
π bug sev-1 cat:performance