Is the change in results significant?

Is the change in results significant?

Is there a significant change- using critical difference

Most communities will see some change in the proportion of children who are developmentally vulnerable, at risk or on track reported in 2015 compared to previous years. In some cases this will be small and in others it will be more substantial. 

Critical difference is the measure that enables tracking of changes over time and calculates whether a change in results between two collections is significant or not. Significant in this case, means a level of certainty that the change did not occur by chance.  

The critical difference calculation takes into account the number of children included in the AEDC data collections and variation between teachers in the way they assess children. 

Number versus proportion

When comparing the proportion of children developmentally vulnerable, at risk or on track across collections, it is important to consider the number and proportion of children.

For smaller communities, results should be interpreted with caution as it only takes a change in a few children to have a big impact on the proportion of children who are developmentally vulnerable, at risk or on track.

Table 1 demonstrates how a slight change in the number of children developmentally vulnerable will have a bigger impact in a small community than in a large community. Community A has 200 children and Community B has 20 children. If 10 per cent of children are vulnerable in both Community A and B, this equates to 20 children in Community A but only two children in Community B. If two more vulnerable children are added to each community, Community A will have 11 per cent of children with developmental vulnerability, and Community B will have 20 per cent of children with developmental vulnerability.​

Table 1: Comparing communities of two sizes

Community

Population

AEDC result (Number)

AEDC result (Proportion)

Addition of 2 vulnerable children to each community

AEDC result (Number)

AEDC result (Proportion)

Community A

200

20

10%

+2

22

11%

Community B

20

2

10%

+2

4

20%

 

Calculating the critical difference

Your ​community may have a different number of children with valid scores, on each of the AEDC domains and summary indicators from one collection to another. If this is so, to calculate the critical difference from the tables below, use the lowest number of children from the two collections you are comparing.

For example, if you would like to know the critical difference of developmental vulnerability in the social competence domain in the ​community of Someplace:

  • the ​community of Someplace had 56 children with a valid score in the social competence domain in 2012 and 81 children in 2015;
  • find the row in the developmentally vulnerable on the social competence domain for a ​community with 56 children (not 81 children);
  • look at the critical difference in that row (the 40-59 children row);

Table 2 shows that Someplace would need to see a change of at least 6.2 percentage points to represent a significant change from 2012 and 2015 in the percentage developmentally vulnerable in the social competence domain. 

Table 2 Developmentally vulnerable critical difference percentage points for the five AEDC domains

 

Critical difference - VULNERABLE

 

SUMMARY INDICATORS

Number of children

Physical Health and Wellbeing

Social Competence

Emotional Maturity

Language and Cognitive skills

Communication and General Knowledge

Vulnerable on 1 or more domains

Vulnerable on 2 or more domains

15-19

14.8

9.9

11.9

10.9

13.2

17.4

12.5

20-24

12.8

8.6

10.3

9.5

11.4

15.1

10.8

25-29

11.5

7.7

9.2

8.5

10.2

13.5

9.7

30-39

10.5

7.1

8.3

7.7

9.3

12.3

8.8

40-59

9.1

6.2

7.2

6.7

8.1

10.7

7.7

60-79

7.5

       5.1

5.8

5.5

6.6

8.7

6.3

80-99

6.5

4.4

5.0

4.7

5.7

7.5

5.4

100-199

5.8

3.9

4.5

4.3

          5.1

6.7

4.9

200-299

4.1

2.8

3.1

3.0

3.6

4.7

3.5

300-699

3.4

       2.3

2.5

2.5

2.9

3.9

2.8

700-1499

2.2

1.5

1.6

1.6

1.9

2.5

1.9

1500-2499

1.5

1.1

1.1

    1.1

1.3

1.7

1.3

2500-3499

1.2

0.8

0.9

0.9

1.0

1.3

1.0

3500-6000

1.0

0.7

0.7

0.7

0.9

1.1

0.8

Table 3. At risk critical difference percentage points for the five AEDC domains

 

Critical difference - ​AT ​RISK

Number of children

Physical Health and Wellbeing

Social Competence

Emotional Maturity

Language and Cognitive skills

Communication and General Knowledge

15-19

18.2

1​6.5

1​7.9

1​7.6

1​9.0

20-24

15.8

1​4.3

15.5

1​5.2

1​6.4

25-29

1​4.1

1​2.8

1​3.8

1​3.6

1​4.7

30-39

12.9

11.7

12.6

12.5

​1​3.​4

40-59

      1​1.2            ​10.2      ​10.9      ​10.8            11.6

60-79

9.2

8.3

8.9

8.8

9.​5

80-99

7.9

7.2

7.7

7.6

​​8.​2

100-199

7.1

6.5

6.9

6.8

7.​4

200-299

5.0

4.6

4.9

4.8

​​5.2

300-699

4.1

3.8

4.0

4.0

​​4.​3

700-1499

2.7

2.5

2.6

2.6

​2.8

1500-2499

1.9

1.7

1.8

1.8

1.9

2500-3499

1.4

1.3

1.4

1.4

1.5

3500-6000

1.2

1.1

1.2

1.2

1.2

 
Table 4. On track critical difference percentage points for the five AEDC domains

 

Critical difference - ON TRACK

Number of children

Physical Health and Wellbeing

Social Competence

Emotional Maturity

Language and Cognitive skills

Communication and General Knowledge

15-19

17.3

12.9

13.5

13.9

15.6

20-24

15.0

11.2

11.7

12.0

13.5

25-29

13.4

10.1

10.5

10.8

12.1

30-39

12.3

9.2

9.6

9.9

​11.​1

40-59

      10.6            8.0      8.4      8.6            9.6

60-79

8.7

6.6

6.9

7.0

​7.​8

80-99

7.6

5.7

6.0

6.1

​6.​8

100-199

6.8

5.1

5.4

5.5

​6.​1

200-299

4.8

3.7

3.8

3.9

​4.​3

300-699

3.9

3.0

3.1

3.2

​3.​5

700-1499

2.6

2.0

2.1

2.1

​2.3

1500-2499

1.8

1.4

1.4

1.5

1.6

2500-3499

1.4

1.1

1.1

1.1

1.2

3500-6000

1.2

0.9

1.0

1.0

1.0


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Further information

A number of resources are available to help understand the concept of critical difference and interpret the AEDC results. These include: