AccumulatorV2 used in RDD











up vote
1
down vote

favorite
1












I have a question about using accumulators in RDD and how to use them reliably.



So imagine we have the following accumulator:



val sc: SparkContext = //...

val accum = sc.longAccumulator("Accumulator name")

val rdd = //some rdd

rdd foreach { _ => 

    accum.add(1L)

}


As far as I understand it the data RDD represents are being splitted by partitions and each time we perform some action it tries to



computeOrReadCheckpoint(split: Partition, context: TaskContext)


for each partition.



So in case we already compute one partition fully we update the accumulator value on the driver side. But after that the executor containing this partition crashes, but the RDD was not checkpointed yet.



So the partition is recomputed from scratch. So I expect the accumulator is updated twice for these records.



Is such a scenario possible?






scala apache-spark







share|improve this question


























    up vote
    1
    down vote

    favorite
    1












    I have a question about using accumulators in RDD and how to use them reliably.



    So imagine we have the following accumulator:



    val sc: SparkContext = //...
    
val accum = sc.longAccumulator("Accumulator name")
    
val rdd = //some rdd
    
rdd foreach { _ => 
    
    accum.add(1L)
    
}


    As far as I understand it the data RDD represents are being splitted by partitions and each time we perform some action it tries to



    computeOrReadCheckpoint(split: Partition, context: TaskContext)


    for each partition.



    So in case we already compute one partition fully we update the accumulator value on the driver side. But after that the executor containing this partition crashes, but the RDD was not checkpointed yet.



    So the partition is recomputed from scratch. So I expect the accumulator is updated twice for these records.



    Is such a scenario possible?






    scala apache-spark







    share|improve this question
























      up vote
      1
      down vote

      favorite
      1









      up vote
      1
      down vote

      favorite
      1






      1





      I have a question about using accumulators in RDD and how to use them reliably.



      So imagine we have the following accumulator:



      val sc: SparkContext = //...
      
val accum = sc.longAccumulator("Accumulator name")
      
val rdd = //some rdd
      
rdd foreach { _ => 
      
    accum.add(1L)
      
}


      As far as I understand it the data RDD represents are being splitted by partitions and each time we perform some action it tries to



      computeOrReadCheckpoint(split: Partition, context: TaskContext)


      for each partition.



      So in case we already compute one partition fully we update the accumulator value on the driver side. But after that the executor containing this partition crashes, but the RDD was not checkpointed yet.



      So the partition is recomputed from scratch. So I expect the accumulator is updated twice for these records.



      Is such a scenario possible?






      scala apache-spark







      share|improve this question













      I have a question about using accumulators in RDD and how to use them reliably.



      So imagine we have the following accumulator:



      val sc: SparkContext = //...
      
val accum = sc.longAccumulator("Accumulator name")
      
val rdd = //some rdd
      
rdd foreach { _ => 
      
    accum.add(1L)
      
}


      As far as I understand it the data RDD represents are being splitted by partitions and each time we perform some action it tries to



      computeOrReadCheckpoint(split: Partition, context: TaskContext)


      for each partition.



      So in case we already compute one partition fully we update the accumulator value on the driver side. But after that the executor containing this partition crashes, but the RDD was not checkpointed yet.



      So the partition is recomputed from scratch. So I expect the accumulator is updated twice for these records.



      Is such a scenario possible?






      scala apache-spark




      scala apache-spark






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Nov 11 at 7:55









      St.Antario

      9,1961550135




      9,1961550135





























          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53246832%2faccumulatorv2-used-in-rdd%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown






























          active

          oldest

          votes













          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.





          Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


          Please pay close attention to the following guidance:


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53246832%2faccumulatorv2-used-in-rdd%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          Full-time equivalent

          Image
          Full-time equivalent From Wikipedia, the free encyclopedia Jump to navigation Jump to search Full-time equivalent ( FTE ) or whole time equivalent ( WTE ) is a unit that indicates the workload of an employed person (or student) in a way that makes workloads or class loads comparable [1] across various contexts. FTE is often used to measure a worker's or student's involvement in a project, or to track cost reductions in an organization. An FTE of 1.0 is equivalent to a full-time worker or student, while an FTE of 0.5 signals half of a full work or school load. [2] Contents 1 U.S. Federal Government 2 In education 2.1 Example 3 Notes 4 References U.S. Federal Government [ edit ] In the U.S. Federal Government, FTE is defined by the Government Accountability Office (GAO) as the number of total hours worked divided by the maximum number of compensable hours in a full-time schedule as
          Read more

          Bicuculline

          Image
          Bicuculline From Wikipedia, the free encyclopedia Jump to navigation Jump to search Bicuculline Clinical data ATC code none Identifiers IUPAC name (6 R )-​6-​[(5 S )-​6-​methyl-​5,​6,​7,​8-​tetrahydro​[1,3]dioxolo​[4,5- g ]​isoquinolin-​5-​yl]​furo[3,4- e ]​[1,3]​benzodioxol-​8(6 H )-​one CAS Number 485-49-4   Y PubChem CID 10237 IUPHAR/BPS 2312 ChemSpider 9820   Y UNII Y37615DVKC ChEBI CHEBI:3092   N ChEMBL ChEMBL417990   N ECHA InfoCard 100.006.927 Chemical and physical data Formula C 20 H 17 N O 6 Molar mass 367.352 g/mol 3D model (JSmol) Interactive image Melting point 215 °C (419 °F) SMILES O=C1O[C@H](c3c1c2OCOc2cc3)[C@@H]5c4cc6OCOc6cc4CCN5C InChI InChI=1S/C20H17NO6/c1-21-5-4-10-6-14-15(25-8-24-14)7-12(10)17(21)18-11-2-3-13-19(26-9-23-13)16(11)20(22)27-18/h2-3,6-7,17-18H,4-5,8-9H2,1H3/t17-,18+/m0/s1   Y Key:IYGYMKDQCDOMRE-ZWKOTPCHSA-N
          Read more

          さくらももこ

          Image
          さくら ももこ 本名 非公開 [1] 生誕 ( 1965-05-08 ) 1965年5月8日 日本・静岡県清水市 （現・静岡市清水区） 死没 ( 2018-08-15 ) 2018年8月15日（53歳没） 日本・東京都目黒区 国籍 日本 職業 漫画家 エッセイスト 作詞家 活動期間 1984年 - 2018年 ジャンル 少女漫画 青年漫画 児童漫画 代表作 『ちびまる子ちゃん』 『コジコジ』 受賞 1989年：第13回講談社漫画賞少女部門 （『ちびまる子ちゃん』） 1990年：第32回日本レコード大賞 （『おどるポンポコリン』作詞） 公式サイト さくらプロダクション テンプレートを表示 さくら ももこ （1965年5月8日 - 2018年8月15日 [2] ）は、日本の漫画家、エッセイスト、作詞家、脚本家。また、自身の少女時代をモデルとした代表作のコミック『ちびまる子ちゃん』の主人公の名前でもある。静岡県清水市（現・静岡市清水区）出身。血液型はA型。身長159cm。一男の母親。 代表作のコミック『ちびまる子ちゃん』の単行本の売上は累計3000万部を超える。また、エッセイストとしても独特の視点と語り口で人気が高く、初期エッセイ集三部作『もものかんづめ』『さるのこしかけ』『たいのおかしら』はいずれもミリオンセラーを記録。 目次 1 経歴 2 人物・交遊関係 3 主な作品 3.1 漫画 3.2 エッセイ 3.3 作詞 3.4 詩集 3.5 雑誌・ムック本 3.6 翻訳 3.7 絵本、ドラマ脚本他 3.8 ラジオ出演 3.9 その他 4 関係人物 4.1 アシスタント 4.2 さくらももこを演じた人物 5 脚注 5.1 注釈 5.2 出典 6 外部リンク 経歴 年譜形式の経歴は推奨されていません 。人物の伝記は流れのあるまとまった文章で記述し、年譜は補助的な使用にとどめてください。 （ 2018年8月 ） 清水市立（当時）入江小学校
          Read more