VMs running kubernetes clusters go down periodically











up vote
0
down vote

favorite












We are running several kubernetes clusters on a few hundred VMs. A few VMs go down every week. We bring it back up. Our metrics show that the CPU & memory usage are low to moderate on these VMs when they go down. Other VM metrics (like the network traffic) also don't point to any unusual patterns. There are no specific messages in /var/log/messages when the VMs go down.



Kubernetes version: 1.9
Linux kernel version: 4.1.12-124.19.5.el7uek.x86_64



Are there other logs or diagnostic information we can check to get to the root cause of the VM outages.






kubernetes virtual-machine







share|improve this question


























    up vote
    0
    down vote

    favorite












    We are running several kubernetes clusters on a few hundred VMs. A few VMs go down every week. We bring it back up. Our metrics show that the CPU & memory usage are low to moderate on these VMs when they go down. Other VM metrics (like the network traffic) also don't point to any unusual patterns. There are no specific messages in /var/log/messages when the VMs go down.



    Kubernetes version: 1.9
    Linux kernel version: 4.1.12-124.19.5.el7uek.x86_64



    Are there other logs or diagnostic information we can check to get to the root cause of the VM outages.






    kubernetes virtual-machine







    share|improve this question
























      up vote
      0
      down vote

      favorite









      up vote
      0
      down vote

      favorite











      We are running several kubernetes clusters on a few hundred VMs. A few VMs go down every week. We bring it back up. Our metrics show that the CPU & memory usage are low to moderate on these VMs when they go down. Other VM metrics (like the network traffic) also don't point to any unusual patterns. There are no specific messages in /var/log/messages when the VMs go down.



      Kubernetes version: 1.9
      Linux kernel version: 4.1.12-124.19.5.el7uek.x86_64



      Are there other logs or diagnostic information we can check to get to the root cause of the VM outages.






      kubernetes virtual-machine







      share|improve this question













      We are running several kubernetes clusters on a few hundred VMs. A few VMs go down every week. We bring it back up. Our metrics show that the CPU & memory usage are low to moderate on these VMs when they go down. Other VM metrics (like the network traffic) also don't point to any unusual patterns. There are no specific messages in /var/log/messages when the VMs go down.



      Kubernetes version: 1.9
      Linux kernel version: 4.1.12-124.19.5.el7uek.x86_64



      Are there other logs or diagnostic information we can check to get to the root cause of the VM outages.






      kubernetes virtual-machine




      kubernetes virtual-machine






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked 2 days ago









      sengs

      2,66662323




      2,66662323
























          1 Answer
          1






          active

          oldest

          votes

















          up vote
          0
          down vote













          Usually we also check the host journal especially if you are running kubelet as systemd.

          There is a good tutorial on digitalocean explaining journald.



          https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs



          It might give you some clue as to why your kube nodes are crashing.






          share|improve this answer





















            Your Answer






            StackExchange.ifUsing("editor", function () {
            StackExchange.using("externalEditor", function () {
            StackExchange.using("snippets", function () {
            StackExchange.snippets.init();
            });
            });
            }, "code-snippets");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "1"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














             

            draft saved


            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53238175%2fvms-running-kubernetes-clusters-go-down-periodically%23new-answer', 'question_page');
            }
            );

            Post as a guest


















            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes








            up vote
            0
            down vote













            Usually we also check the host journal especially if you are running kubelet as systemd.

            There is a good tutorial on digitalocean explaining journald.



            https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs



            It might give you some clue as to why your kube nodes are crashing.






            share|improve this answer

























              up vote
              0
              down vote













              Usually we also check the host journal especially if you are running kubelet as systemd.

              There is a good tutorial on digitalocean explaining journald.



              https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs



              It might give you some clue as to why your kube nodes are crashing.






              share|improve this answer























                up vote
                0
                down vote










                up vote
                0
                down vote









                Usually we also check the host journal especially if you are running kubelet as systemd.

                There is a good tutorial on digitalocean explaining journald.



                https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs



                It might give you some clue as to why your kube nodes are crashing.






                share|improve this answer












                Usually we also check the host journal especially if you are running kubelet as systemd.

                There is a good tutorial on digitalocean explaining journald.



                https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs



                It might give you some clue as to why your kube nodes are crashing.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered yesterday









                Bal Chua

                36215




                36215






























                     

                    draft saved


                    draft discarded



















































                     


                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53238175%2fvms-running-kubernetes-clusters-go-down-periodically%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest






































































                    -

                    Popular posts from this blog

                    Full-time equivalent

                    Image
                    Full-time equivalent From Wikipedia, the free encyclopedia Jump to navigation Jump to search Full-time equivalent ( FTE ) or whole time equivalent ( WTE ) is a unit that indicates the workload of an employed person (or student) in a way that makes workloads or class loads comparable [1] across various contexts. FTE is often used to measure a worker's or student's involvement in a project, or to track cost reductions in an organization. An FTE of 1.0 is equivalent to a full-time worker or student, while an FTE of 0.5 signals half of a full work or school load. [2] Contents 1 U.S. Federal Government 2 In education 2.1 Example 3 Notes 4 References U.S. Federal Government [ edit ] In the U.S. Federal Government, FTE is defined by the Government Accountability Office (GAO) as the number of total hours worked divided by the maximum number of compensable hours in a full-time schedule as
                    Read more

                    さくらももこ

                    Image
                    さくら ももこ 本名 非公開 [1] 生誕 ( 1965-05-08 ) 1965年5月8日 日本・静岡県清水市 （現・静岡市清水区） 死没 ( 2018-08-15 ) 2018年8月15日（53歳没） 日本・東京都目黒区 国籍 日本 職業 漫画家 エッセイスト 作詞家 活動期間 1984年 - 2018年 ジャンル 少女漫画 青年漫画 児童漫画 代表作 『ちびまる子ちゃん』 『コジコジ』 受賞 1989年：第13回講談社漫画賞少女部門 （『ちびまる子ちゃん』） 1990年：第32回日本レコード大賞 （『おどるポンポコリン』作詞） 公式サイト さくらプロダクション テンプレートを表示 さくら ももこ （1965年5月8日 - 2018年8月15日 [2] ）は、日本の漫画家、エッセイスト、作詞家、脚本家。また、自身の少女時代をモデルとした代表作のコミック『ちびまる子ちゃん』の主人公の名前でもある。静岡県清水市（現・静岡市清水区）出身。血液型はA型。身長159cm。一男の母親。 代表作のコミック『ちびまる子ちゃん』の単行本の売上は累計3000万部を超える。また、エッセイストとしても独特の視点と語り口で人気が高く、初期エッセイ集三部作『もものかんづめ』『さるのこしかけ』『たいのおかしら』はいずれもミリオンセラーを記録。 目次 1 経歴 2 人物・交遊関係 3 主な作品 3.1 漫画 3.2 エッセイ 3.3 作詞 3.4 詩集 3.5 雑誌・ムック本 3.6 翻訳 3.7 絵本、ドラマ脚本他 3.8 ラジオ出演 3.9 その他 4 関係人物 4.1 アシスタント 4.2 さくらももこを演じた人物 5 脚注 5.1 注釈 5.2 出典 6 外部リンク 経歴 年譜形式の経歴は推奨されていません 。人物の伝記は流れのあるまとまった文章で記述し、年譜は補助的な使用にとどめてください。 （ 2018年8月 ） 清水市立（当時）入江小学校
                    Read more

                    13 indicted, 8 arrested in Calif. drug cartel investigation

                    Image
                    Fox News U.S. World Opinion Politics Entertainment Business Lifestyle TV Radio More Expand / Collapse search Login Watch TV ☰ Hot Topics U.S. Crime Military Education Terror Immigration Economy Personal Freedoms World U.N. Conflicts Terrorism Disasters Global Economy Environment Religion Scandals Opinion Politics Executive Senate House Judiciary Foreign policy Polls Elections Entertainment Celebrity News Movies TV News Music News Style News Entertainment Video Business Markets Politics Technology Features Business Leaders Lifestyle Food + Drink Cars + Trucks Travel + Outdoors House + Home Fitness + Well-being Style + Beauty Family Science Archaeology Air & Space Planet Earth Wild Nature Natural Science Dinosaurs Tech Security Innovation Drones Computers Video Games Military Tech Health Healthy Living Medical Research Mental Health Cancer Heart Health Children'
                    Read more