Stack Exchange
Running a Wildfly 16 instance with a jmx prometheus javaagent and logging to a graylog server
I want to setup a wildfly 16 instance with surveillance via jmx prometheus javaagent and logging via graylog configured in the standalone.xml file.
I had success running the jmx prometheus javaagent with following settings:

(standalone.conf)
lmPath=$(find $JBOSS_HOME/modules -name "jboss-logmanager*.jar" | xargs -I {} readlink -m "{}")
commonPath=$(find $JBOSS_HOME/modules -name "wildfly-common-*.jar" | xargs -I {} readlink -m "{}")

JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS,org.jboss.logmanager"
JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"
JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$lmPath"
JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$commonPath"
JAVA_OPTS="$JAVA_OPTS -javaagent:/opt/jmx/jmx_prometheus_javaagent.jar=12345:/opt/jmx/wildfly-10.yaml"

I also had success running the logging to the graylog server with following settings:

(standalone.conf)
JAVA_OPTS="$JAVA_OPTS -Djboss.server.log.dir=/var/log/wildfly"
JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS"
JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"
JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$JBOSS_HOME/jboss-modules.jar"

(standalone.xml)
            <custom-handler name="GREYLOG2" class="org.graylog2.log.GelfAppender" module="org.graylog2.logging">
                <level name="INFO"/>
                <properties>
                    <property name="graylogHost" value="tcp:127.0.0.1"/>
                    <property name="graylogPort" value="5555"/>
                    <property name="extractStacktrace" value="true"/>
                    <property name="facility" value="gelf-java"/>
                    <property name="originHost" value="myHost" />
                    <property name="additionalFields" value="{'environment': 'DEV', 'application': 'myApp'}" />
                </properties>
            </custom-handler>  
            <root-logger>
                <level name="INFO"/>
                <handlers>
                    <handler name="CONSOLE"/>
                    <handler name="GREYLOG2"/>
                </handlers>
            </root-logger>

I included the needed modules.
So basicly both on its own run fine but iam not able to start the instance with both features. My assumed configuration looks like this:
lmPath=$(find $JBOSS_HOME/modules -name "jboss-logmanager*.jar" | xargs -I {} readlink -m "{}")
commonPath=$(find $JBOSS_HOME/modules -name "wildfly-common-*.jar" | xargs -I {} readlink -m "{}")

JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS,org.jboss.logmanager"
JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"
JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$lmPath"
JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$commonPath"
JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$JBOSS_HOME/jboss-modules.jar"
JAVA_OPTS="$JAVA_OPTS -javaagent:/opt/jmx/jmx_prometheus_javaagent.jar=12345:/opt/jmx/wildfly-10.yaml"
JAVA_OPTS="$JAVA_OPTS -Djboss.server.log.dir=/var/log/wildfly"

Same standalone.xml as in the graylog case.
This throws following exception:
Failed to read or configure the org.jboss.logmanager.LogManager
java.lang.IllegalArgumentException: Failed to load module "org.jboss.as.logging" for handler "GREYLOG2"
    at org.jboss.logmanager.config.AbstractPropertyConfiguration.<init>(AbstractPropertyConfiguration.java:65)
    at org.jboss.logmanager.config.HandlerConfigurationImpl.<init>(HandlerConfigurationImpl.java:51)
    at org.jboss.logmanager.config.LogContextConfigurationImpl.addHandlerConfiguration(LogContextConfigurationImpl.java:143)
    at org.jboss.logmanager.PropertyConfigurator.configureHandler(PropertyConfigurator.java:673)
    at org.jboss.logmanager.PropertyConfigurator.configureLogger(PropertyConfigurator.java:583)
    at org.jboss.logmanager.PropertyConfigurator.configure(PropertyConfigurator.java:521)
    at org.jboss.logmanager.PropertyConfigurator.configure(PropertyConfigurator.java:97)
    at org.jboss.logmanager.LogManager.readConfiguration(LogManager.java:170)
    at org.jboss.logmanager.LogManager.readConfiguration(LogManager.java:132)
    at java.util.logging.LogManager$3.run(LogManager.java:399)
    at java.util.logging.LogManager$3.run(LogManager.java:396)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396)
    at java.util.logging.LogManager.access$800(LogManager.java:145)
    at java.util.logging.LogManager$2.run(LogManager.java:345)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338)
    at java.util.logging.LogManager.getLogManager(LogManager.java:378)
    at java.util.logging.Logger.demandLogger(Logger.java:448)
    at java.util.logging.Logger.getLogger(Logger.java:502)
    at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.<clinit>(JmxCollector.java:38)
    at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:29)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
    at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: org.jboss.modules.ModuleNotFoundException: org.jboss.as.logging
    at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:294)
    at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:280)
    at org.jboss.logmanager.config.AbstractPropertyConfiguration$ModuleFinder.getClassLoader(AbstractPropertyConfiguration.java:560)
    at org.jboss.logmanager.config.AbstractPropertyConfiguration.<init>(AbstractPropertyConfiguration.java:63)
    ... 27 more
org.jboss.modules.ModuleNotFoundException: org.jboss.as.standalone
    at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:294)
    at org.jboss.modules.Main.main(Main.java:437)


update:
the file standalone/configuration/logging.properties is automatically altered on startup and now contains the following:
handler.GREYLOG2=org.jboss.as.logging.logmanager.Log4jAppenderHandler
handler.GREYLOG2.module=org.jboss.as.logging
handler.GREYLOG2.level=INFO
handler.GREYLOG2.formatter=GREYLOG2
handler.GREYLOG2.postConfiguration=activate
handler.GREYLOG2.properties=appender,enabled,dummy
handler.GREYLOG2.appender=GREYLOG2
handler.GREYLOG2.enabled=true
handler.GREYLOG2.dummy=


Is there more to the stack trace? Given the above configuration I'm not sure where the org.jboss.as.logging module would be coming from. That said you likely also need org.wildfly.common on the $JBOSS_MODULES_SYSTEM_PKGS as well.

Hi and thank you James. Thats all output. I tried adding org.wildfly.common to $JBOSS_MODULES_SYSTEM_PKGS but it doesnt change anything.

Ah I see the issue now. You're hitting https://issues.redhat.com/browse/WFCORE-4674. At this point there is probably not a workaround that I can think of.
