菜单

Explanation of Query Logs

Published 2024年11月06日

Scenario Description

This article will explain how to query logs for various components of JumpServer.

Operation Instructions

The default logs are already mounted in the persistent directory, and you can also view them directly in the persistent directory.

JumpServer v4 version, the default persistent directory is /data/jumpserver, and it can also be queried with the command: cat /opt/jumpserver/config/config.txt | grep "VOLUME_DIR"

ls -al /data/jumpserver/core/logs
ls -al /data/jumpserver/koko/data/logs
ls -al /data/jumpserver/lion/data/logs
ls -al /data/jumpserver/nginx/data/logs

Core

The Core component is the core component of JumpServer, and other components depend on it for startup.

docker logs -f jms_core --tail 200
# If you need to operate within the container.
docker exec -it jms_core bash
cd /opt/jumpserver/data/logs
ls -al

total 8860
drwxr-xr-x. 9 root root    4096  3月 15 23:59 .
drwxr-xr-x. 1 root root      29  3月  2 17:08 ..
drwxr-xr-x. 2 root root     112  3月  9 23:59 2023-03-09  # Historical logs are split by day.
drwxr-xr-x. 2 root root     112  3月 10 23:59 2023-03-10
drwxr-xr-x. 2 root root     112  3月 11 23:59 2023-03-11
drwxr-xr-x. 2 root root     112  3月 12 23:59 2023-03-12
drwxr-xr-x. 2 root root     112  3月 13 23:59 2023-03-13
drwxr-xr-x. 2 root root     112  3月 14 23:59 2023-03-14
drwxr-xr-x. 2 root root     112  3月 15 23:59 2023-03-15
-rw-r--r--. 1 root root       0  2月  3 11:03 ansible.log
-rw-r--r--. 1 root root  109899  3月 16 16:22 beat.log
-rw-r--r--. 1 root root   24716  3月 16 14:15 celery_ansible.log
-rw-r--r--. 1 root root  344414  3月 16 16:22 celery_default.log
-rw-r--r--. 1 root root       1  3月 16 02:00 celery.log
-rw-r--r--. 1 root root   34788  2月 28 10:45 daphne.log
-rw-r--r--. 1 root root   12502  2月 28 10:40 drf_exception.log
-rw-r--r--. 1 root root       0  3月 15 23:59 flower.log
-rw-r--r--. 1 root root 1934510  3月 16 16:22 gunicorn.log
-rw-r--r--. 1 root root 5774260  3月 16 14:00 jumpserver.log  # The core log is primarily focused on this.
-rw-r--r--. 1 root root  273249  3月 16 11:29 unexpected_exception.log
# If there are no exceptions, you can also check other logs for anomalies; pay attention to the log timestamps.
tail -f jumpserver.log -n 200
# When sending logs to others for troubleshooting, ensure that you provide the complete logs. Refer to this section:
2023-03-16 11:29:32 [log ERROR] Internal Server Error: /api/v1/accounts/accounts/su-from-accounts/            #  <---- Make sure to include the start time.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 486, in thread_handler
    raise exc_info[1]
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 38, in inner
    response = await get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 233, in _get_response_async
    response = await wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 448, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
 
···
 
  File "/opt/jumpserver/apps/accounts/api/account/account.py", line 47, in su_from_accounts
    accounts = self.filter_queryset(accounts)
  File "/opt/jumpserver/apps/common/api/filter.py", line 35, in filter_queryset
    queryset = backend().filter_queryset(self.request, queryset, self)
  File "/usr/local/lib/python3.9/site-packages/django_filters/rest_framework/backends.py", line 90, in filter_queryset
    filterset = self.get_filterset(request, queryset, view)
  File "/usr/local/lib/python3.9/site-packages/django_filters/rest_framework/backends.py", line 31, in get_filterset
    filterset_class = self.get_filterset_class(view, queryset)
  File "/usr/local/lib/python3.9/site-packages/django_filters/rest_framework/backends.py", line 64, in get_filterset_class
    assert issubclass(queryset.model, filterset_model), \
AttributeError: 'list' object has no attribute 'model'                              #  <---- Some users may only send this one line, which is incorrect.
2023-03-16 11:46:32 [connection ERROR] Unsubscribe msg error: 'NoneType' object has no attribute 'clear_connect_callbacks' #  <---- All errors between this time and the next must be sent in full.
 
# When sending diagnostic logs to others, please follow this rule: if multiple errors occur within the same time frame, send them completely based on the timestamps.
# If there are duplicate logs, please deduplicate them first.

Koko

Koko is a component that serves Unix-like operating systems, providing character-based connections via SSH and Telnet protocols.

docker logs -f jms_koko --tail 200
# If you need to operate within the container.
docker exec -it jms_koko bash
cd /opt/koko/data/logs
ls -al
total 216
drw-------. 2 root root     22  2月 22 11:07 .
drwxr-xr-x. 6 root root     58  2月  3 11:07 ..
-rw-r--r--. 1 root root 220476  3月 16 16:33 koko.log    # koko log
tail -f koko.log -n 200
# koko log
2023-03-16 16:21:22 [ERRO] Ws[cd4a6c4f-5cc3-450b-a2fd-cbda9415b0ae] read data err: websocket: close 1005 (no status)
2023-03-16 16:21:32 [ERRO] Ws[24d1ae87-291a-4c90-8dac-8af59df60e1d] read data err: websocket: close 1001 (going away)
2023-03-16 16:21:32 [ERRO] Session[dc6946af-e8ff-498b-8bf7-a1d2bdb9bd40] user read err: io: read/write on closed pipe
2023-03-16 16:21:32 [ERRO] Session[dc6946af-e8ff-498b-8bf7-a1d2bdb9bd40] srv read err: EOF
2023-03-16 16:21:35 [ERRO] Ws[24d1ae87-291a-4c90-8dac-8af59df60e1d] send CLOSE message err: websocket: close sent
2023-03-16 16:21:39 [ERRO] Ws[40ab251b-5cd3-4baa-bbf2-fe18f35c3b16] read data err: websocket: close 1006 (abnormal closure): unexpected EOF
2023-03-16 16:21:39 [ERRO] Ws[40ab251b-5cd3-4baa-bbf2-fe18f35c3b16] send CONNECT message err: write tcp 192.168.250.8:5000->192.168.250.6:36540: use of closed network connection
2023-03-16 16:22:08 [ERRO] Get new ssh client err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023-03-16 16:22:08 [ERRO] ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023-03-16 16:22:08 [ERRO] Start connecting to root(ssh key)@10.1.13.27 error: Authentication failed (username or password incorrect)

Lion

Lion is a component that serves the Windows system platform, used for web access to Windows assets.

docker logs -f jms_lion --tail 200
# If you need to operate within the container.
docker exec -it jms_lion bash
cd /opt/lion/data/logs
ls -al
total 116
drwxr-xr-x. 2 root root     39 Feb 22 22:23 .
drwxr-xr-x. 7 root root     71 Feb  3 11:07 ..
-rw-r--r--. 1 root root   3499 Mar 15 22:13 guacd.log
-rw-r--r--. 1 root root 113517 Mar 16 16:30 lion.log     # lion log
tail -f lion.log -n 200
# lion log
2023-03-15 22:44:33 tunnel conn.go [ERROR] Session[0eb6c532-73b2-42a4-b6ca-94e789d68a15] receive web client disconnect opcode
2023-03-15 22:44:33 tunnel conn.go [ERROR] Session[0eb6c532-73b2-42a4-b6ca-94e789d68a15] web client read err: websocket: close 1005 (no status)
2023-03-15 22:44:33 tunnel conn.go [ERROR] Session[0eb6c532-73b2-42a4-b6ca-94e789d68a15] guacamole server read err: EOF
2023-03-15 22:45:30 tunnel conn.go [ERROR] Session[72e804c3-be37-409e-b857-cbde83b41a0a] receive web client disconnect opcode
2023-03-15 22:45:30 tunnel conn.go [ERROR] Session[72e804c3-be37-409e-b857-cbde83b41a0a] web client read err: websocket: close 1005 (no status)
2023-03-15 22:45:30 tunnel conn.go [ERROR] Session[72e804c3-be37-409e-b857-cbde83b41a0a] send web client err: websocket: close sent
2023-03-15 22:45:31 tunnel conn.go [ERROR] Session[dfd382ad-533b-4e6b-9ba6-f1ec4c6bdb23] receive web client disconnect opcode
2023-03-15 22:46:29 tunnel conn.go [ERROR] Session[b8df0c15-d2e7-4b27-a56d-5d66108d2f0d] web client read err: websocket: close 1005 (no status)
2023-03-15 22:46:29 tunnel conn.go [ERROR] Session[b8df0c15-d2e7-4b27-a56d-5d66108d2f0d] guacamole server read err: EOF
2023-03-16 15:39:16 main main.go [ERROR] Ws client read err: websocket: close 1006 (abnormal closure): unexpected EOF
2023-03-16 15:39:16 main main.go [ERROR] Ws heart beat closed, try reconnect after 10s
2023-03-16 16:30:40 tunnel conn.go [ERROR] Session[e5262183-e959-441d-ba1a-34c29733c1fe] receive web client disconnect opcode
2023-03-16 16:30:40 tunnel conn.go [ERROR] Session[e5262183-e959-441d-ba1a-34c29733c1fe] web client read err: websocket: close 1005 (no status)
2023-03-16 16:30:40 tunnel conn.go [ERROR] Session[e5262183-e959-441d-ba1a-34c29733c1fe] send web client err: websocket: close sent

Web

The Web component is used to provide the frontend interface of JumpServer.

docker logs -f jms_web --tail 200
# If you need to operate within the container.
docker exec -it jms_web sh
cd /var/log/nginx
ls -al
total 8776
drwxr-xr-x. 2 root  root    4096 Mar 16 06:25 .
drwxr-xr-x. 1 root  root      70 Mar  2 17:11 ..
-rw-r-----. 1 nginx adm  1700469 Mar 16 16:45 access.log
-rw-r-----. 1 nginx adm  4235610 Mar 16 06:24 access.log.1
-rw-r-----. 1 nginx adm        0 Mar 16 06:25 error.log
-rw-r-----. 1 nginx adm     3773 Mar 16 06:25 error.log.1
-rw-r--r--. 1 nginx root       0 Feb  3 11:07 tcp-access.log
tail -f error.log -n 200
# nginx log
2023/03/15 20:03:52 [warn] 58#58: *174753 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/4/05/0000000054 while reading upstream, client: 10.1.10.35, server: , request: "GET /core/flower/static/js/jquery-ui-1-8-15.min.js?v=947f1df57c47a338b395e07e3f09b17b7088577ed958bd64e8519a6579cd252d14e60d78f42786164faa2fa51265c0ba9ef517ccee583c9d37603a671dbbf95d HTTP/1.1", upstream: "http://192.168.250.3:8080/core/flower/static/js/jquery-ui-1-8-15.min.js?v=947f1df57c47a338b395e07e3f09b17b7088577ed958bd64e8519a6579cd252d14e60d78f42786164faa2fa51265c0ba9ef517ccee583c9d37603a671dbbf95d", host: "xxx.xxx.com", referrer: "https://xxx.xxx.com/core/flower/?_=1678881720571"
2023/03/15 20:03:52 [warn] 61#61: *174793 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/5/05/0000000055 while reading upstream, client: 10.1.10.35, server: , request: "GET /core/flower/static/js/moment-timezone-with-data.min.js?v=4762b13071ca18e6c83105de008e54f6009a43d849e103d14c06535564066d9c7984d95233c631393a3c476c2bd4931c5f031d8e56eccaa18a08348e409f3b4d HTTP/1.1", upstream: "http://192.168.250.3:8080/core/flower/static/js/moment-timezone-with-data.min.js?v=4762b13071ca18e6c83105de008e54f6009a43d849e103d14c06535564066d9c7984d95233c631393a3c476c2bd4931c5f031d8e56eccaa18a08348e409f3b4d", host: "xxx.xxx.com", referrer: "https://xxx.xxx.com/core/flower/?_=1678881720571"

Celery

Celery is a component that handles asynchronous tasks, used to execute automation tasks related to JumpServer.

# If you need to operate within the container.
docker exec -it jms_celery bash
cd /opt/jumpserver/logs
ls -al
total 8772
drwxr-xr-x. 9 root root    4096  3月 15 23:59 .
drwxr-xr-x. 1 root root      29  3月  2 17:08 ..
drwxr-xr-x. 2 root root     112  3月  9 23:59 2023-03-09
drwxr-xr-x. 2 root root     112  3月 10 23:59 2023-03-10
drwxr-xr-x. 2 root root     112  3月 11 23:59 2023-03-11
drwxr-xr-x. 2 root root     112  3月 12 23:59 2023-03-12
drwxr-xr-x. 2 root root     112  3月 13 23:59 2023-03-13
drwxr-xr-x. 2 root root     112  3月 14 23:59 2023-03-14
drwxr-xr-x. 2 root root     112  3月 15 23:59 2023-03-15
-rw-r--r--. 1 root root       0  2月  3 11:03 ansible.log
-rw-r--r--. 1 root root  110838  3月 16 16:31 beat.log
-rw-r--r--. 1 root root   24716  3月 16 14:15 celery_ansible.log            # The Celery logs can be found under the logs that start with "celery_".
-rw-r--r--. 1 root root  347225  3月 16 16:31 celery_default.log
-rw-r--r--. 1 root root       1  3月 16 02:00 celery.log
-rw-r--r--. 1 root root   34788  2月 28 10:45 daphne.log
-rw-r--r--. 1 root root   12502  2月 28 10:40 drf_exception.log
-rw-r--r--. 1 root root       0  3月 15 23:59 flower.log
-rw-r--r--. 1 root root 2191794  3月 16 16:31 gunicorn.log
-rw-r--r--. 1 root root 5774365  3月 16 16:28 jumpserver.log
-rw-r--r--. 1 root root  273249  3月 16 11:29 unexpected_exception.log
tail -f celery_default.log -n 200
# celery log
>> Set language to zh
>> Set org to 00000000-0000-0000-0000-000000000000
Task settings.tasks.ldap.import_ldap_user[d716aaa1-2c7b-40e7-a6da-346e6875034f] succeeded in 0.5366005189716816s: None
Task settings.tasks.ldap.import_ldap_user[e0fd1b82-3d28-433a-994d-cb075684d396] received
>> Set language to zh
>> Set org to 00000000-0000-0000-0000-000000000000
Task settings.tasks.ldap.import_ldap_user[e0fd1b82-3d28-433a-994d-cb075684d396] succeeded in 0.5023798840120435s: None
Task settings.tasks.ldap.import_ldap_user[193e3682-7968-4cec-827a-ced3082dcdc2] received
>> Set language to zh
>> Set org to 00000000-0000-0000-0000-000000000000
Task settings.tasks.ldap.import_ldap_user[193e3682-7968-4cec-827a-ced3082dcdc2] succeeded in 0.5524193355813622s: None
Task settings.tasks.ldap.import_ldap_user[67191c1d-e779-4b37-a67e-ca78c3217610] received
>> Set language to zh
>> Set org to 00000000-0000-0000-0000-000000000000
Task settings.tasks.ldap.import_ldap_user[67191c1d-e779-4b37-a67e-ca78c3217610] succeeded in 0.3798262616619468s: None

Previous Flexible Solutions for Switching to Root Privileges
Next Enhancing Server Security with Login Approval Mechanisms