Let’s resolve into factors

a⁶ + 3a³b³-40b⁶

(a³)² + 3a³b³-40(b³)²

To resolve it into factors we first resolve 40

40=2×2×2×5

Given, x² + (m + n)x + mn=(x + m)(x + n) … (2)

Comparing 1 and 2

m + n =3

mn =-40

∴ (1) can be written as

(a³)² + 8a³b³-5a³b³-40(b³)²

= (a³ + 8b³)( a³-5b³)

∴ The resolved factors are (a³ + 8b³)( a³-5b³)

Apply the formula a³ + b³ = (a+b)(a² + b² – ab) in (a³ + 8b³)

Now (a³ + 8b³) = (a³ + (2b)³)

= (a² + 4b² – 2ab)( a³-5b³)

∴ The resolved factors are (a² + 4b² – 2ab)( a³-5b³).