blob: 6374da736f998ae82c46dfea27c4a5adefea90f8 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
|
Sep 24 13:34:18 ipekatrin2 kernel: mongod invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=999
Sep 24 13:34:18 ipekatrin2 kernel: mongod cpuset=docker-225d08e93d9cf3516fabb7605bc3a61739dd3a580015576ca3c0cf804d43364e.scope mems_allowed=0-1
Sep 24 13:34:18 ipekatrin2 kernel: CPU: 16 PID: 54317 Comm: mongod Tainted: G W ------------ 3.10.0-693.el7.x86_64 #1
Sep 24 13:34:18 ipekatrin2 kernel: Hardware name: Supermicro X10DRH/X10DRH-CT, BIOS 2.0a 06/30/2016
Sep 24 13:34:18 ipekatrin2 kernel: ffff881f84a79fa0 000000009693f540 ffff8821f85dfcb8 ffffffff816a3d91
Sep 24 13:34:18 ipekatrin2 kernel: ffff8821f85dfd48 ffffffff8169f186 ffff8828e3aab280 0000000000000001
Sep 24 13:34:18 ipekatrin2 kernel: 0000000000000000 0000000000000000 ffff8821f85dfcf8 0000000000000046
Sep 24 13:34:18 ipekatrin2 kernel: Call Trace:
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff816a3d91>] dump_stack+0x19/0x1b
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff8169f186>] dump_header+0x90/0x229
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff81185ee6>] ? find_lock_task_mm+0x56/0xc0
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff811f1578>] ? try_get_mem_cgroup_from_mm+0x28/0x60
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff81186394>] oom_kill_process+0x254/0x3d0
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff811f5296>] mem_cgroup_oom_synchronize+0x546/0x570
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff811f4710>] ? mem_cgroup_charge_common+0xc0/0xc0
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff81186c24>] pagefault_out_of_memory+0x14/0x90
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff8169d54e>] mm_fault_error+0x68/0x12b
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff816b01f1>] __do_page_fault+0x391/0x450
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff816b02e5>] do_page_fault+0x35/0x90
Sep 24 13:34:18 ipekatrin2 kernel: [<ffffffff816ac508>] page_fault+0x28/0x30
Sep 24 13:34:18 ipekatrin2 kernel: Task in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb350130e_ac45_11e9_bbd6_0cc47adef0e6.slice/docker-225d08e93d9cf3516fabb7605bc3a61739dd3a580015576ca3c0cf804d43364e.scope killed as a result of limit of /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb350130e_ac45_11e9_bbd6_0cc47adef0e6.slice
Sep 24 13:34:18 ipekatrin2 kernel: memory: usage 524160kB, limit 524288kB, failcnt 7138692017
Sep 24 13:34:18 ipekatrin2 kernel: memory+swap: usage 524164kB, limit 9007199254740988kB, failcnt 0
Sep 24 13:34:18 ipekatrin2 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
Sep 24 13:34:18 ipekatrin2 kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb350130e_ac45_11e9_bbd6_0cc47adef0e6.slice: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Sep 24 13:34:18 ipekatrin2 kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb350130e_ac45_11e9_bbd6_0cc47adef0e6.slice/docker-5c123505863e871160334c30fc02a40dc363775313e62339b1e4b1cf479d1c79.scope: cache:0KB rss:544KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:544KB inactive_file:0KB active_file:0KB unevictable:0KB
Sep 24 13:34:18 ipekatrin2 kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb350130e_ac45_11e9_bbd6_0cc47adef0e6.slice/docker-225d08e93d9cf3516fabb7605bc3a61739dd3a580015576ca3c0cf804d43364e.scope: cache:580KB rss:522740KB rss_huge:0KB mapped_file:548KB swap:0KB inactive_anon:0KB active_anon:522740KB inactive_file:0KB active_file:92KB unevictable:0KB
Sep 24 13:34:18 ipekatrin2 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Sep 24 13:34:18 ipekatrin2 kernel: [62561] 1001 62561 671 120 7 0 -998 pod
Sep 24 13:34:18 ipekatrin2 kernel: [53610] 1000160000 53610 219890 128941 358 0 999 mongod
Sep 24 13:34:18 ipekatrin2 kernel: [81470] 1000160000 81470 42932 1775 62 0 999 mongo
Sep 24 13:34:18 ipekatrin2 kernel: Memory cgroup out of memory: Kill process 57372 (mongod) score 1984 or sacrifice child
Sep 24 13:34:18 ipekatrin2 kernel: Killed process 53610 (mongod) total-vm:879560kB, anon-rss:515764kB, file-rss:0kB, shmem-rss:0kB
Sep 24 13:34:18 ipekatrin2 origin-node: sh: cannot set terminal process group (-1): Inappropriate ioctl for device
Sep 24 13:34:18 ipekatrin2 origin-node: sh: no job control in this shell
Sep 25 07:56:01 ipekatrin2 atomic-openshift-master-api: E0925 07:56:01.269000 32946 writers.go:70] error encountered while streaming results via websocket: unexpected EOF
Sep 25 07:56:01 ipekatrin2 origin-node: E0925 07:56:01.428012 93115 kuberuntime_logs.go:222] Failed with err write tcp 192.168.13.2:10250->192.168.13.2:56684: i/o timeout when writing log for log file "/var/log/pods/2e6fec81-1486-11e9-bbd6-0cc47adef0e6/kdb-backend_1.log": &{timestamp:{sec:63704994961 nsec:201763176 loc:<nil>} stream:stdout log:[50 48 49 57 45 48 57 45 50 53 32 32 55 58 53 54 58 48 49 32 49 52 48 52 49 54 53 48 56 48 54 53 53 51 54 32 91 78 111 116 101 93 32 69 118 101 110 116 32 83 99 104 101 100 117 108 101 114 58 32 80 117 114 103 105 110 103 32 116 104 101 32 113 117 101 117 101 46 32 48 32 101 118 101 110 116 115 10]}
Sep 25 07:56:04 ipekatrin2 origin-node: sh: no job control in this shell
Sep 25 07:56:04 ipekatrin2 origin-node: ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
Sep 25 08:07:41 ipekatrin2 systemd-logind: Failed to start session scope session-118144.scope: Connection timed out
Sep 25 08:08:05 ipekatrin2 systemd-logind: Failed to start session scope session-118145.scope: Connection timed out
Sep 25 08:08:07 ipekatrin2 systemd: Failed to start Session 118145 of user root.
Sep 25 08:08:07 ipekatrin2 systemd: Failed to start Session 118144 of user root.
Sep 26 08:54:14 ipekatrin2 etcd: lost the TCP streaming connection with peer 2696c5f68f35c672 (stream MsgApp v2 reader)
Sep 26 08:54:14 ipekatrin2 etcd: failed to read 2696c5f68f35c672 on stream MsgApp v2 (read tcp 192.168.13.2:33022->192.168.13.3:2380: i/o timeout)
Sep 26 08:54:14 ipekatrin2 etcd: peer 2696c5f68f35c672 became inactive
Sep 26 08:54:14 ipekatrin2 etcd: lost the TCP streaming connection with peer 2696c5f68f35c672 (stream Message reader)
Sep 26 08:54:57 ipekatrin2 etcd: lost the TCP streaming connection with peer 2696c5f68f35c672 (stream MsgApp v2 writer)
Sep 26 08:54:57 ipekatrin2 etcd: peer 2696c5f68f35c672 became active
Sep 26 08:54:57 ipekatrin2 etcd: established a TCP streaming connection with peer 2696c5f68f35c672 (stream Message reader)
Sep 26 08:54:57 ipekatrin2 etcd: established a TCP streaming connection with peer 2696c5f68f35c672 (stream MsgApp v2 reader)
Sep 26 08:55:00 ipekatrin2 atomic-openshift-master-api: E0926 08:55:00.567709 32946 controller.go:162] unable to sync kubernetes service: Service "kubernetes" is invalid: spec.clusterIP: Invalid value: "172.30.0.1": provided IP is already allocated
Sep 26 08:55:02 ipekatrin2 etcd: established a TCP streaming connection with peer 2696c5f68f35c672 (stream MsgApp v2 writer)
Sep 26 08:55:10 ipekatrin2 atomic-openshift-master-api: W0926 08:55:10.581936 32946 lease_endpoint_reconciler.go:176] Resetting endpoints for master service "kubernetes" to [192.168.13.1 192.168.13.2]
Sep 26 08:56:51 ipekatrin2 kernel: CIFS VFS: open dir failed
Sep 26 09:00:01 ipekatrin2 sh: 66f56f307485
Sep 26 09:34:37 ipekatrin2 origin-node: sh: no job control in this shell
Sep 26 09:34:37 ipekatrin2 origin-node: ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
Sep 26 09:34:44 ipekatrin2 origin-node: sh: no job control in this shell
Sep 26 09:34:44 ipekatrin2 origin-node: ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
Sep 26 18:44:48 ipekatrin2 etcd: apply entries took too long [228.552929ms for 1 entries]
Sep 26 18:44:48 ipekatrin2 etcd: avoid queries with large range/delete range!
|
