Uploaded image for project: 'Apache Spark'
  1. Apache Spark
  2. SPARK-1171

when executor is removed, we should reduce totalCores instead of just freeCores on that executor

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      When the executor is removed, the current implementation will only reduce the freeCores on that executor. Actually we should reduce the totalCores...

      The impact of this bug is that the default parallelism of a job may be set incorrectly after an executor is removed (so an RDD may get split into more partitions than the amount of parallelism in the cluster). In other words, the bug leads to performance degredation but no correctness problem.

        Gliffy Diagrams

          Activity

          Hide
          CodingCat Nan Zhu added a comment -
          Show
          CodingCat Nan Zhu added a comment - proposed a PR: https://github.com/apache/spark/pull/63

            People

            • Assignee:
              CodingCat Nan Zhu
              Reporter:
              CodingCat Nan Zhu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: