Beyond Modularization: Scaling your Android Build with Gradle

Transcript

1. Beyond Modularization: Scaling Android Builds Nelson Osacky

2. Me • Previously Android Engineer • Large projects • SoundCloud

   • Square • Small startups • Solutions Engineer at Gradle

3. Me • Gradle Plugin Maintainer • Fladle - Easily Scale

   Instrumentation Tests on Firebase https://github.com/runningcode/fladle • Delect - Replace Dagger with Dagger Reflect https://github.com/soundcloud/delect/ • Gradle Doctor - Actionable Insights for your build https://github.com/runningcode/gradle-doctor

4. Solutions Engineer https://gradle.com/enterprise/trial/

5. None

6. Gradle Build Tool Gradle Enteprise

7. Android Gradle Plugin Android Studio

8. Kotlin Intellij

9. Android Development Feature Development Tech Debt (There are other things

   too)

10. Tech Debt Refactoring Build Speed

11. Build Speed is Tech Debt And it always pays off

12. Cost of Builds 60s waste * 50 builds / day

    * 50 devs = 42 hours lost / day

13. Cost of Builds 60s waste * 50 builds / day

    * 50 devs = 42 hours lost / day not including lost focus https://gradle.com/roi-calculator

14. Fast Builds Matter

15. Cost of Builds 60s waste * 50 builds / day

    * 50 devs = 42 hours lost / day hire 5 new people without paying them! no recruiting https://gradle.com/roi-calculator

16. Build Speed is Tech Debt And it always pays off

    And is easy to justify working on it

17. Motivated? Where do we start?

18. None

19. Update All The Things ⬢ Gradle 6.5 ⬢ Android Gradle

    Plugin 4.0.0 ⬢ Gradle Enterprise Plugin 3.3.4 ⬢ Kotlin 1.3.72 ⬢ Third party plugins ⬢ Third party libraries New Performance APIs Caching Improvements Task Configuration Avoidance Background Scan Uploads Incremental Annotation Processors Compiler Perf Improvements

20. Update All The Things plugins { id "com.github.ben-manes.versions" version "0.28.0"

    } ./gradlew dependencyUpdates -Drevision=release https://github.com/ben-manes/gradle-versions-plugin
21. None

22. Build Lifecycle •Initialization •Configuration •Execution

23. Build Lifecycle Initialization Sets up the environment for the build

    and determines which projects will take part in it.

24. Build Lifecycle Configuration All build scripts of all projects are

    executed. Constructs and configures the task graph.

25. Build Lifecycle Execution Runs the tasks graph in order.

26. Where to start?

27. Check for Red Flags

28. What is a red flag?

29. If nothing changes, no tasks should execute.

30. ./gradlew assembleDebug ./gradlew assembleDebug BUILD SUCCESSFUL in 2s 58 actionable

    tasks: 58 up-to-date

31. ./gradlew assembleDebug ./gradlew assembleDebug BUILD SUCCESSFUL in 5s 359 actionable

    tasks: 3 executed, 356 from cache

32. Check for Red Flags Regularly

33. BUILD SUCCESSFUL in 5s 359 actionable tasks: 3 executed, 356

    up-to-date Publishing build scan... https://gradle.com/s/yj5jtd4kh6ucc Build Analyzer results available

34. BUILD SUCCESSFUL in 5s 359 actionable tasks: 3 executed, 356

    up-to-date Publishing build scan... https://gradle.com/s/yj5jtd4kh6ucc Build Analyzer results available Gradle Build Scans Android Studio Build Analyzer

35. Android Studio Build Analyzer

36. Run a build scan ./gradlew assembleDebug --scan https://scans.gradle.com/

37. None

38. Filter

39. None
40. None

41. 9 tasks ran Bugsnag

42. None
43. None

44. #perfmatters

45. None
46. None

47. 1 sec per project is high

48. None

49. 25 sec is red flag!

50. apply plugin: 'witness'

51. //apply plugin: 'witness'

52. None

53. 5.8s to 0.5s by disabling two plugins!

54. Configuration time savings affect every build

55. Configuration time should be roughly 5s per 100 projects

56. Disable plugins for local builds

57. engBuild pattern

58. engBuild != buildVariant

59. engBuild == Gradle property https://docs.gradle.org/current/userguide/build_environment.html#sec:gradle_configuration_properties

60. enabled by default ext.engBuild = (project.findProperty('engBuild') ?: 'true') .toBoolean()

61. if (!engBuild) { apply plugin: 'com.google.firebase.crashlytics' apply plugin: 'com.google.firebase.firebase-perf' }

62. ./gradlew assembleDebug ./gradlew assembleRelease -PengBuild=false disable on CI or testing

    release builds

63. engBuild=false disable on CI or testing release builds gradle.properties

64. engBuild pattern

65. 16627 tasks is a red flag

66. Use Task Configuration Avoidance https://docs.gradle.org/current/userguide/task_configuration_avoidance.html

67. api vs implementation

68. Module A Module B Module C dependencies { implementation project('module-b')

    implementation project('module-c') } dependencies { implementation project('module-c') }

69. Module A Module B Module C dependencies { implementation project('module-b')

    } dependencies { api project('module-c') } module-c must always used when consuming module-b

70. api vs implementation convenience vs performance

71. Real World Example

72. https://github.com/k9mail/k-9 K9 Mail App

73. K9 Mail App https://github.com/k9mail/k-9/blob/master/app/core/build.gradle#L12 apiaproject(':plugins:openpgp-api-lib:openpgp-api')

74. K9 Mail App https://github.com/k9mail/k-9/blob/master/app/core/build.gradle#L12 apiaproject(':plugins:openpgp-api-lib:openpgp-api')

75. apiaproject(':plugins:openpgp-api-lib:openpgp-api') K9 Mail App

76. if (project.hasProperty('useImpl')) { implementation project(':plugins:openpgp-api-lib:openpgp-api') } else { apiaproject(':plugins:openpgp-api-lib:openpgp-api') }

    K9 Mail App

77. ./gradlew :app:k9mail:assembleDebug ./gradlew :app:k9mail:assembleDebug -PuseImpl K9 Mail App

78. Profile it!

79. https://github.com/gradle/gradle-profiler Gradle Profiler

80. Let's write scenarios Gradle Profiler

81. change_api_dep { } performance.scenarios Gradle Profiler

82. change_api_dep { title = "Change using api dependency" } performance.scenarios

    Gradle Profiler

83. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] } performance.scenarios Gradle Profiler

84. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler

85. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler

86. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } performance.scenarios Gradle Profiler

87. change_impl_dep { title = "Change using impl dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler

88. change_impl_dep { title = "Change using impl dependency" tasks =

    ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler

89. change_impl_dep { title = "Change using impl dependency" tasks =

    ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler

90. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } performance.scenarios Gradle Profiler

91. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } performance.scenarios Gradle Profiler

92. gradle-profiler --benchmark --scenario-file performance.scenarios Gradle Profiler

93. 1.stop daemon 2.clean 3.6 warm up builds 4.10 benchmark builds

    Gradle Profiler
94. None

95. html report and csv report

96. Build Time (milliseconds) 0 3250 6500 9750 13000 #1 #2

    #3 #4 #5 #6 #7 #8 #9 #10 Change using api dependency Change using impl dependency

97. scenario Change using api dependency Change using impl dependency mean

    11603.0 10151.7 median 11529.5 10049.5 confidence 0.0 99.98429477157690

98. ~15% improvement

99. why?

100. gradle-profiler --profile buildscan --scenario-file performance.scenarios

101. gradle-profiler --profile buildscan --scenario-file performance.scenarios * Results written to /Users/no/workspace/k-9/profile-out-14

     Scenario Change using api dependency using Gradle 6.5 - Build scan for measured build #1: https://gradle.com/s/ozev5czodllsk Scenario Change using impl dependency using Gradle 6.5 - Build scan for measured build #1: https://gradle.com/s/rxujhuyxqlybu

102. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency

103. https://scans.gradle.com/s/rxujhuyxqlybu/timeline?outcomeFilter=SUCCESS implementation Dependency

104. Which tasks ran?

105. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency

106. https://scans.gradle.com/s/rxujhuyxqlybu/timeline?outcomeFilter=SUCCESS implementation Dependency

107. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency

108. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency

109. What exactly is the difference?

110. Enterprise Feature Alert

111. Capture Task Input Files https://docs.gradle.com/enterprise/gradle-plugin/#capturing_task_input_files

112. gradleEnterprise { buildScan { captureTaskInputFiles = true } } https://docs.gradle.com/enterprise/gradle-plugin/#capturing_task_input_files

113. Build Scan Comparison

114. Build Scan Comparison

115. Build Scan Comparison Build B uses api

116. Adding unnecessary inputs to tasks.

117. How do we fix it?

118. https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin Dependency Analysis Plugin plugins { id "com.autonomousapps.dependency-analysis" version "0.49.0"

     }
119. None

120. Annotation Processors

121. Make sure they are incremental!

122. w: [kapt] Incremental annotation processing requested, but support is disabled

     because the following processors are not incremental: androidx.room.RoomProcessor (NON_INCREMENTAL).

123. android { ... defaultConfig { ... javaCompileOptions { annotationProcessorOptions {

     arguments += [ "room.schemaLocation":"$projectDir/schemas".toString(), "room.incremental":"true"] } } } }

124. Room

125. android { ... defaultConfig { ... javaCompileOptions { annotationProcessorOptions {

     arguments += [ "room.schemaLocation":"$projectDir/schemas".toString(), "room.incremental":"true"] } } } }

126. "room.schemaLocation":"$projectDir/schemas".toString(), Remote Cache miss!

127. "room.schemaLocation":"$projectDir/schemas".toString(), Remote Cache miss! https://developers.soundcloud.com/blog/gradle-remote-build-cache-misses

128. Move Room to it's own Room https://github.com/gradle/android-cache-fix-gradle-plugin Module

129. None

130. Disabling unused AGP Features

131. Cost of Generating BuildConfig All 186 are UP-TO-DATE.

132. Unused so Proguard will remove it later

133. BuildConfig adds java sources to pure kotlin modules

134. Which BuildConfig?

135. Which version code?

136. Which version code?

137. VERSION_CODE is 1 by default in library modules

138. Aidl what?

139. Also adds task configuration time.

140. None

141. android.defaults.buildfeatures.buildconfig=false android.defaults.buildfeatures.aidl=false android.defaults.buildfeatures.renderscript=false android.defaults.buildfeatures.resvalues=false android.defaults.buildfeatures.shaders=false gradle.properties New in AGP 4.0.0

     https://developer.android.com/studio/releases/gradle-plugin#buildFeatures

142. Disable for all projects

143. Enable when needed

144. android.buildFeatures { buildConfig = true } app/build.gradle

145. Code Multipliers

146. AndroidX Jetifier

147. Migrates support-library- dependent libraries to rely on Android X packages.

148. android.enableJetifier=true

149. None

150. android.enableJetifier=false

151. Dependency Substitution https://issuetracker.google.com/issues/156449751

152. Stuck on the Jetifier?

153. plugins { id "com.github.plnice.canidropjetifier" version "0.5" } Can I drop

     Jetifier? ./gradlew -Pandroid.enableJetifier=false canIDropJetifier

154. Jetify At Home ./jetifier-standalone -i <source-library> -o <output-library> https://developer.android.com/studio/command-line/jetifier

155. Jetify At Home Upload to nexus

156. Flavors

157. Flavor Explosion

158. None

159. Flavor Explosion When you add a build dimension multiplying build

     speed in a surprising surprise.

160. Flavor Explosion BuildType * BuildFlavor = BuildVariants

161. Flavor Explosion BuildType * BuildFlavor = BuildVariants BuildVariant are code

     multipliers

162. Flavor Explosion 2 BuildTypes * 3 BuildFlavors = 6 Variants

     6x code 6x tasks 6x resources 6x cache utilization

163. def prop = project.findProperty("enabledBuildVariant") ?: "freeDebug" android.variantFilter { variant ->

     if (variant.name != prop) { variant.ignore = true } } ./gradlew assemblePaidDebug -PenableBuildVariant=paidDebug

164. Flavor Explosion • Mostly a configuration time win. • Permanently

     disable flavors you don't need.

165. How to cut build times in half

166. How to cut build times in half.

167. apply plugin: 'com.android.library'

168. assembleDebug assembleRelease

169. 2x code 2x resources 2x tasks 2x tests !!!!!

170. How often does the debug and release code differ for

     library modules?

171. ./gradlew test runs all tests twice

172. ./gradlew testDebug java module tests don't run

173. Recompile everything when switching to release

174. CI takes twice as long

175. if (engBuild) { } Disable unused flavors in library modules

176. if (engBuild) { android.variantFilter { variant -> } } Disable

     unused flavors in library modules

177. if (engBuild) { android.variantFilter { variant -> if (variant.name ==

     'release') { variant.ignore = true } } } Disable unused flavors in library modules

178. if (engBuild) { android.variantFilter { variant -> if (variant.name ==

     'release') { variant.ignore = true } } } Disable unused flavors in library modules

179. if (engBuild) { android.variantFilter { variant -> if (variant.name ==

     'release') { variant.ignore = true } } } Disable unused flavors in library modules

180. if (engBuild) { android.variantFilter { variant -> if (variant.name ==

     'debug') { variant.ignore = true } } } Disable unused flavors in library modules

181. android.variantFilter { variant -> if (variant.name == 'debug') { variant.ignore

     = true } } Disable unused flavors in library modules android.debug.matchingFallbacks = ['release'] in application module

182. android.variantFilter { variant -> if (variant.name == 'debug') { variant.ignore

     = true } } Disable unused flavors in library modules

183. Build and test times cut in half

184. Switching to release build doesn't recompile everything

185. Disable stuff! #protip

186. Disable stuff! #protip measure!

187. ./gradlew help Measure configuration time

188. Disable debug variant Disable debug variant and AGP features Disable

     AGP features Standard build 0.0 200.0 400.0 600.0 800.0 Avg Configuration Time (ms) Average of 10 builds help using Gradle Profiler

189. 783ms to 688ms ~= 13% faster

190. 13% configuration time improvement that scales

191. Regular build assembleDebug

192. assembleDebug Debug disabled, AGP features disabled

193. Disable More Things https://github.com/android/gradle-recipes

194. engBuild Ideas

195. engBuild Ideas •Higher minSdk •Disables Multidex •Disable coreLibraryDesugaring •Strip other

     languages •Strip large images •Strip out native libraries •Try new versions of AGP https://medium.com/@runningcode/testing-new-versions-of-the-android-gradle-plugin-ea80df978316

196. Always Measure!

197. Upcoming Performance Enhancements

198. File System Watching

199. Fast input snapshotting

200. File System Watching https://blog.gradle.org/introducing-file-system-watching org.gradle.unsafe.watch-fs=true

201. Configuration Caching

202. Configuration Caching

203. Configuration Caching The build scripts of all projects which are

     part of the build are executed.

204. ext.engBuild = (project.findProperty('engBuild') ?: 'true') .toBoolean()

205. ext.engBuild = providers.gradleProperty("engBuild") .forUseAtConfigurationTime() .orElse("true") .map { it.toBoolean() }

206. if (!engBuild) { apply plugin: 'com.google.firebase.crashlytics' apply plugin: 'com.google.firebase.firebase-perf' }

207. if (!engBuild.get()) { apply plugin: 'com.google.firebase.crashlytics' apply plugin: 'com.google.firebase.firebase-perf' }

208. Improving Android Build Performance Devoxx Belgium 2018 Cédric Champeau https://www.youtube.com/watch?v=StahHZ-L83k

     https://speakerdeck.com/melix/improving-android-build-performance

209. Build Bigger, Better: Gradle for Large Projects Google I/O 2019

     Aurimas Liutikas and Xavier Ducrohet https://www.youtube.com/watch?v=sQC9-Rj2yLI

210. The Secrets of the Build Scan Plugin and the Internals

     of Gradle Virtual Android Makers Paris 2020 Me https://www.youtube.com/watch?v=lgaqS0pmUzk

211. Measuring remote build cache performance https://medium.com/@runningcode https://github.com/runningcode/gradle-doctor

212. Thank you! Need advice? osacky.com https://gradle.com/enterprise/trial/ https://speakerdeck.com/runningcode/beyond-modularization-scaling-your-android-build-with-gradle